SC12 - NASA Research Exhibit - SC12 Salt Lake City, UT 

http://scl2.supercomputing.org 

November 10-16, 2012 

AERONAUTICS (ARMD) 

Demo 1 

A Breakthrough in Rotorcraft Prediction Accuracy Using Detached Eddy Simulation 

Overview 

Helicopters and tiltrotors provide many useful civil and military functions without the need 
for airports and runways. Modern rotorcraft designs continue to push the technology to 
improve vehicle performance, safety, and reduce its impact on the environment. However, 
the accurate simulation of rotorcraft flow fields with computational fluid dynamics (CFD) 
continues to be a challenging problem. Rotor blades generate blade-tip vortices as they 
spin around and encounter the vortices of other blades as well. This can result in very 
complex blade vortex interaction (BVI) and vortex wake interactions. Moreover, rotorcraft 
simulation is multidisciplinary and must take into account the interaction between rotor 
blade aerodynamics, blade flexibility, and blade motions to achieve trimmed steady flight. 

Project Details 

The figure of merit (FM), the chief hover performance parameter, is under predicted by 2- 
6% depending on the numerical approach. To bring this into perspective, every Vz% error 
means there is one less passenger the vehicle can carry. Moreover, under resolved rotor 
wakes result in inaccurate interactions between the rotor vortices and the rest of the 
vehicle. One goal of NASA’s Rotary Wing (RW) Project is to develop physics-based 
computational tools to improve the predictive accuracy of these flow fields. The 
OVERFLOW CFD code is used to solve the time-dependent Navier-Stokes equations for the 
V22 Osprey tiltrotor and UH-60 Blackhawk helicopter rotors in hover and forward flight. 

Results and Impact 

The FM has been predicted within experimental error for the first time for the V22 Osprey 
and Blackhawk helicopter rotors in hover over a wide range of flow conditions. This 
dramatic improvement is due to the use of algorithms with high-order spatially accuracy, 
refined grid resolution on the rotor blades, and an improved detached eddy simulation 
(DES) hybrid turbulence model. High resolution accuracy on the rotor blades was 
important for all cases studied, but the DES hybrid turbulence model was crucial in BVI 
cases. 

Dynamic adaptive mesh refinement (AMR) is used to efficiently resolve the rotor wakes. 
This is accomplished by detecting vortices and locally added blocks of Cartesian grids of 
higher resolution. This allows the vortices and rotor wake to be resolved by grids four 
times finer and at half the computational cost than a uniform (non AMR) grid refinement 
approach. The AMR resolution revealed tremendous detail of the blade vortices and 
turbulent flow that had never before been observed computationally or experimentally. 

The flow simulations have been quantitatively validated with experimental measurements. 


The use of AMR greatly improved the prediction of vortex core size and growth with wake 
age (distance along the tip vortex). The error in vortex core growth rate has been reduced 
from 60% to 4% error. 

These groundbreaking results will impact the next generation of rotorcraft analysis and 
design. 

Role of High-End Computing Resources and Services 

These flow simulations were made possible by NASA’s Pleiades supercomputer. Solutions 
typically required 1,500-4,600 cores for 1-2 weeks. Grid sizes varied from 60-750 million 
grid points, and required 5-24 hours of wall time per rotor revolution. 

Researcher and Presenter atSC12: 

Neal M. Chaderjian, NASA Ames Research Center 
Neal.m.chaderjian@nasa.gov 

Demo 2 

Adjoint-Based Design for Complex Aerospace Configurations 

Overview 

This effort seeks to enable formal constrained design optimization of complex aerospace 
configurations experiencing unsteady turbulent flows and requiring dynamic/deforming 
geometries and high-fidelity physical models. Since a single simulation of this type may 
require millions of compute hours, highly efficient strategies must be developed to 
compute accurate sensitivities for many thousands of design parameters. Adjoint-based 
methods implemented on massively parallel computing architectures allow the designer to 
formally tackle a broad range of complex optimization problems across the speed range 
which would otherwise remain computationally intractable. The overall approach is 
implemented in the NASA Langley FUN3D solver 

Project Details 

In a typical fluid dynamics problem, the governing equations are integrated forward in 
physical time to determine aerodynamic quantities of interest. To efficiently compute 
sensitivities of these quantities with respect to thousands of input parameters, the adjoint 
approach solves the linearized governing equations in reverse physical time. In this 
manner, the adjoint solution marches backwards in time to determine the sensitivities of 
output functions due to any input parameters required for the simulation. 

A discrete adjoint approach is taken in which the discretized governing equations are 
linearized exactly with respect to the flowfield and mesh variables. The implementation 
supports steady and unsteady flows based on compressible and incompressible forms of 
the Reynolds-averaged Navier-Stokes equations. Computational meshes may include 
general element types belonging to single block or overset grid topologies. Furthermore, 
these grids may be rigidly moving, deforming, or any combination thereof; general parent- 
child body motion relationships are also accommodated. 

Results and Impact 

The effort required to develop and mature this general capability has taken two decades of 
sustained research and software development. The approach has been successfully 



demonstrated on a broad range of realistic design problems, including rotorcraft 
geometries, fighter jets, biologically-inspired micro air vehicle configurations, active flow 
control systems, and wind energy devices. The adjoint-based methodology also provides a 
rigorous foundation, valuable insight, and an effective tool for error estimation, mesh 
adaptation, and uncertainty quantification for such problems. 

Role of High-End Computing Resources and Services 

A relatively simple single-point design optimization typically requires the equivalent cost 
of 0(30) flow simulations. Since a single simulation for this class of flows may require 
millions of compute hours, efficiently exploiting the world’s most powerful 
supercomputing systems is essential. Furthermore, solution of the adjoint equations 
requires that the complete forward solution be available during the reverse integration in 
time. This translates to terabytes of data I/O for even moderately-sized problems; 
development of a highly efficient asynchronous parallel I/O strategy has been an enabling 
element of this work. 

Researcher and Presenter atSC12: 

Eric Nielsen, NASA Langley Research Center 
Eric.J.Nielsen@nasa.gov 

Demo 3 

Simulating Hypersonic Turbulent Combustion for Future Aircraft 

Overview 

Future access-to-space activities and transoceanic air travel may rely on advanced aero- 
propulsion concepts to enable hypersonic flight (Mach numbers greater than 5). A 
hydrogen-fueled scramjet (supersonic combusting ramjet) is one type of engine capable of 
hypersonic flight, and the analysis, design, and optimization of such engines relies heavily 
on computational fluid dynamics. A more accurate way of computing the reactive flow 
within scramjet engines is through large-eddy simulation (LES), which directly calculates 
the larger turbulent eddies responsible for mixing of fuel and air. In this work, we conduct 
LES simulations of the reacting flow within a model scramjet tested at the University of 
Virginia. 

Project Details 

For this project, we solve the compressible Navier-Stokes equations governing a reacting 
mixture of gases using a large-eddy simulation strategy. Finite-rate hydrogen-air 
combustion is modeled through a nine-species / nineteen reaction mechanism. The 
equations are advanced in time using an implicit method. Low-dissipation discretization 
schemes are used to allow resolution of small turbulent eddies. The calculations use 
meshes ranging size from 35 to 70 million cells and were performed using NCSU’s 
REACTMB flow solver. Results were compared with experimental data collected at the 
University of Virginia. 

Results and Impact 

Our research shows that flame stabilization in the University of Virginia’s model scramjet 
combustor is a complex process, dependent on the interplay between the relatively fast 



hydrogen oxidation kinetics, high mixing rates induced by the flameholder geometry, and 
flame stretch effects induced by large-eddy interaction with the flame surface. The LES 
methodology provides much better predictive capability, compared with Reynolds- 
averaged Navier-Stokes methods that are in popular use. Excellent agreement with 
experimental temperature, mole fraction, velocity, and pressure measurements have been 
obtained. 

Role of High-End Computing Resources and Services 

These simulations simultaneously solve upwards of 500 million equations and have been 
performed on hundreds of processors simultaneously on NASA’s Pleiades supercomputer. 
Each simulation requires several days to complete and generates from 4 to 8 terabytes of 
data. 

Researcher: Jack Edwards, North Carolina State University 
jredward@ncsu.edu 

Presenter atSC12: Jesse Fulton, North Carolina State University 
jafuiton@ncsu.edu 


Demo 4 

From a Roar to a Whisper: Making Modern Aircraft Quieter 

Overview 

Aircraft noise is an environmental hazard that affects metropolitan areas adjacent to major 
airports. With the projected growth in air travel, the noise exposure levels are expected to 
increase substantially. The Environmentally Responsible Aviation (ERA) project of the 
NASA Aeronautics Mission Research Directorate is developing and maturing noise 
reduction technologies that will confine aircraft noise footprints on the ground within 
airport boundaries. This is to be accomplished without detrimental effects to the 
aerodynamic efficiency of aircraft. 

To most people, aircraft noise is synonymous with jet engine noise. During approach to 
landing, however, noise generated by the airframe is comparable to, and in some instances 
even louder than, propulsion noise. The prominent sources of airframe noise are the 
landing gear and lift generating devices such as wing flaps and slats. 

Project Details 

As part of the ERA-Gulfstream partnership on airframe noise research, we simulated the 
airflow over a semi-span model of a Gulfstream aircraft in landing configuration. We 
focused on accurately capturing the prominent noise sources at both flap tips for the 
baseline configuration. Then, we evaluated the effectiveness of a novel noise reduction 
concept applied to the flap tips. The time-accurate simulations were obtained using the 
NASA Langley FUN3D unstructured Navier-Stokes flow solver and the Exa Corporation 
PowerFLOW Lattice-Boltzmann flow solver. Approximately one billion volumetric cells 
were used to discretize the space surrounding the aircraft surfaces. 


Results and Impact 



Our simulations have provided unique and detailed insight into the prominent fluid 
dynamical processes that create noise at the flap tips. Scrutiny of the flow field in the 
vicinity of the tips revealed the formation/presence of multiple vortices that behave like 
miniature tornados. Their evolution, merging, and interaction with the sharp corners at the 
tips produce high-amplitude sound waves that propagate towards the ground. We also 
demonstrated that the noise abatement concept applied to the tips is very effective. 

Role of High-End Computing Resources and Services 

The vast number of arithmetic operations needed to perform any of our simulations 
requires usage of the largest supercomputers. All the simulations were performed on 
NASA’s Pleiades supercomputer using approximately two thousand processors. Depending 
on the nature of the needed information, the computational runs could last from two weeks 
to two months, resulting in several Petabytes of data. The rendering and effective 
visualization of such massive datasets were made possible by the members of the NAS 
visualization group at NASA Ames. 

Researcher and Presenter atSC12: 

Mehdi R. Khorrami, NASA Langley Research Center 
mehdi.r.khorrami@nasa.gov 

Demo 5 

Modeling of Extended Formation Flight on High-Performance Computers 

Overview 

As in avian flight, aircraft flown in formation can obtain significant savings in induced drag 
due to favorable upwash of the lead aircraft. As the world’s fleet of aircraft grows, 
formation flight is one approach toward reducing fuel consumption on long-range flights. 
NASA’s Fundamental Aeronautics project is studying this concept as one wedge in a 
comprehensive strategy toward reduced carbon emissions and fuel consumption. 

While formation flight has been studied since the early 1900’s, estimates of the potential 
benefits have varied significantly. Credible estimates of induced drag savings of 10-30% 
have been predicted by low-order methods but have been difficult to confirm with high- 
fidelity simulations. 

Project Details 

Even with modern computing hardware, these simulations pose a significant challenge 
since they hinge on accurately predicting and measuring induced drag of complete aircraft 
at transonic conditions and accurately propagating vortices and wakes over long distances 
through a computational domain. In extended formation flight, operational safety dictates 
separation distances of 15-50 spans, further increasing the level of difficulty. 

These challenges are met by using a non-body-fitted simulation approach on meshes which 
dynamically adapt to minimize errors in the measurement of induced drag of the aircraft in 
formation. The simulations include the effects of flow compressibility, aircraft trim, control 
surface deflection and other important physics which are lacking in previous work. 


Results and Impact 



Results from these simulations are verified with flight-test data being run cooperatively 
with other government agencies. After validation, these data are used in models of the 
National Airspace System to quantify the potential of formation flight to reduce fuel 
consumption at a national scale. 

Role of High-End Computing Resources 

Formation flight involves aerodynamic phenomena spanning lengths from millimeters to 
miles. This extraordinary range of scales and the sensitivity of the output to error in the 
simulation make these extremely challenging problems. With problem sizes that are on the 
order of a billion degrees of freedom, and hundreds of cases to consider, the availability of 
NASA’s high-end computing resources are the key to this research. 

Researcher and Presenter atSC12: 

Michael Aftosmis, NASA Ames Research Center 
Michael.Aftosmis@nasa.gov 


HUMAN EXPLORATION OPERATIONS fHEOMDl 
Demo 6 

Supersonic Retropropulsion for Mars Entry 

Overview 

Supersonic Retropropulsion (SRP) is a potential method of decelerating high mass vehicles 
entering the Martian atmosphere. This technology, which employs jets opposing a 
supersonic freestream, is needed because current methods of supersonic deceleration do 
not scale to human exploration or large robotic sized spacecraft. Computational Fluid 
Dynamics (CFD) is important to the development of SRP because flight and ground testing 
is difficult and cost-prohibitive at these flow conditions. To validate CFD at these conditions, 
three NASA flow solvers were employed and two wind tunnel tests were conducted 
allowing code-to-code and code-to-test comparisons. The comparisons are positive and 
show that CFD is capable of simulating SRP. 

Project Details 

Three NASA-developed Navier-Stokes flow solvers were used in a CFD validation effort for 
SRP. The codes are Data Parallel Line Relaxation (DPLR), Fully Unstructured Navier-Stokes 
Three-Dimensional (FUN3D), and OVERset grid FLOW solver (OVERFLOW). Two wind 
tunnel tests were completed with the specific purpose of CFD validation. The tests were 
conducted in the Langley and Ames Unitary Plan Wind Tunnels, and produced surface 
pressure data as well as high speed flow visualization imagery. Qualitative and quantitative 
comparisons between codes and test data show a high capability of CFD to simulate SRP 
flowfields. 

Results and Impact 

The capability of CFD to simulate SRP flowfields will allow CFD to be used for parametric 
studies, vehicle control and stability analysis, and properly predicting the complex physics 
of SRP at Martian entry conditions. 


Role of High-End Computing Resources and Services 

The CFD validation studies required time-accurate simulations, which used millions of 
processor hours and created terabytes of data. NASA supercomputing resources not only 
made an analysis of this size possible, but also enabled the project to complete the 
computations efficiently in a timely manner. 

Researcher and Presenter at SC12: 

Guy Schauerhamer, Jacobs/ NASA Johnson Space Center 
Daniel.G.Schauerhamer@nasa.gov 

Demo 7 

Validating Water Spray Simulation Models for the SLS Launch Environment 

Overview 

NASA regularly uses computational fluid dynamics (CFD) simulations to assess the loads 
and risks for space vehicles at liftoff. Traditionally, these simulations have been used to 
make predictions either near the vehicle, such as for plume recirculation, or for large scale 
effects like wind. However, validations against recent test data have shown that modern 
codes have the capability to make meaningful pressure predictions over the full launchpad 
for frequencies up to 150 Hz. This limit is high enough that significant phenomena, like the 
transient startup and overpressure pulse of rockets, can be modeled. 

While the results of these validations have allowed CFD to make immediate impacts on the 
design of NASA's new Space Launch System, they have also been primarily for dry 
launchpads. To examine the full range of designs, CFD also needs to incorporate multi- 
phase components like water sprays. To enable this work, such a capability was developed 
in partnership with Mississippi State, and then used to simulate the effects of water 
suppression for a second validation case. 

Project Details 

For this project, the emission, flight, and evaporation of water spray systems on an active 
launchpad was simulated using a two-phase approach of lagrangian water particles and gas 
phase CFD elements. The simulation was based on a test case from the Ares Scale Model 
Acoustic Test so that pressure predictions from across the launchpad could be validated 
against real data. Approximately 200 million cells of resolution were used to capture the 
fine detail and geometry of the pad, and millions of particles were tracked to capture the 
dynamics of a water cascade. This work was performed using a density-based finite- 
volume CFD program called Loci/CHEM. 

Results and Impact 

Ignition overpressure and payload acoustics are design considerations which are 
consistently risk drivers for new launch vehicles. Traditionally, these effects have been 
predicted using analytical methods, with low-fidelity scaling of results from old designs. 
However, the technology development and validation efforts supported by this work are 
allowing CFD to make validated predictions of pressure over a full launchpad using the 
geometry as designed for a specific vehicle. Rather than relying solely on "rules of thumb” 
to guide suppression strategies, such systems can be optimized for maximum effect using 



CFD parametrics Furthermore, problems with these systems can be identified and fixed 
before metal is ever cut, providing significant cost savings to NASA. 


Role of High-End Computing Resources and Services 

Prediction of acoustic effects with CFD for a structure the size of a launch pad is an intense 
computational task, and one which would be extremely difficult without the resources of a 
supercomputer like Pleiades to support it. Simulations of even a scaled rocket took nearly 
a week to complete while running on over 1000 processors, and were normally executed in 
groups of two or more to test design permutations. In addition, the storage requirements 
for such simulations were on the order of terabytes per simulation, with regular processing 
of hundreds of individual gigabyte scale files necessary for visualization, post-processing, 
and analysis of data. 

Researcher and Presenter atSC12: 

Gabriel Putnam, NASA Marshall Space Flight Center / APL 
gabriel.c.putnam@nasa.gov 

Demo 8 

Simulating Moving Valves for Space Launch System Liquid Engines 

Overview 

Three-dimensional, time accurate fluid dynamics analysis of moving valves offers 
tremendous insight for valve design and operation. Fluid dynamics that occur during valve 
actuation often have significant impact on the valve and connected hardware. However, 
due to a lack of computational resources, fluid dynamic analysis of transient valve motion 
is rarely performed as part of a valve design process. 

In some valve applications, such as the Space Shuttle Orbiter Flow Control Valve (FCV), the 
first sign of significant fluid dynamics caused by valve actuation was the fluid induced high 
cycle fatigue failure of the valve poppet. Computational Fluid Dynamics analysis performed 
at Marshall Space Flight Center using the density-based fluid analysis program LOCI\Chem 
succeeded in identifying fluid mode shapes and amplitudes that could have possibly caused 
the FCV to fail. The analysis indicated that the exact conditions that led to the valve failure 
probably occurred at some point during the valve actuation. However, due to limitations of 
the CFD tools and processing power available in 2009, all analysis was performed with the 
valve in a fixed, static position. The inconclusive results of the static position FCV analysis 
and the associated delays in the Orbiter flight schedule caused by the valve failure 
emphasized the real need to develop the ability to perform time-accurate computational 
fluid dynamics analysis of moving valves. 

Marshall Space Flight Center has developed the ability to model large-scale moving valve 
simulations using LOCI\Chem. Due to the efficient parallelization of LOCI\Chem and the 
increased computational resources available at NASA’s Ames Research Facility, moving 
valve simulations are completed quickly enough to support valve design cycles. 

Project Details 

For this project, the opening and closing cycle of a candidate Space Launch System FCV is 
modeled to evaluate potentially damaging unsteady fluid environments in the valve and 
other critical performance metrics. Simulations were conducted using LOCI\Chem with an 



overset mesh method to simulate the moving valve components. A hybrid unstructured 
mesh with ~55 million cells was required to resolve the fluid dynamics of interest. A hybrid 
Reynolds-Averaged Navier-Stokes/ Large Eddy Simulation (hybrid RANS-LES) turbulence 
model was used to accurately capture time-accurate pressure oscillations in the flow. 

Results and Impact 

CFD analysis of the moving valve identified the onset of a periodic pressure response in the 
flow which only occurs after the valve re-opens from closure. The analysis showed that the 
tool holes downstream of the flow orifices were contributing to a 14Khz pressure 
oscillation in the flow. A recommendation to fill the tool hole with an insert was given to 
the valve vendor. The vendor recognized the benefit of eliminating this source of unsteady 
fluid environments and incorporated the recommendation into the design. The analysis 
also quantified the unsteady forces on the poppet head due to pressure changes during 
actuation. 

Role of High-End Computing Resources and Services 

In order for the CFD analysis to provide input for the valve design cycle simulations were 
massively parallelized on NASA’s Pleiades supercomputer. Simulations were performed 
using approximately 600 processors in order to complete one closing and re-opening cycle 
in a working week. 

Researcher and Presenter atSC12: 

Brian R. Richardson, Qualis Corp., NASA/Marshall Space Flight Center 
brian.r.richardson@nasa.gov 

Demo 9 

Innovative Simulations for Modeling the SLS Solid Rocket Booster Ignition 

Overview 

NASA’s new heavy launch vehicle, known as the Space Launch System (SLS), will utilize 2 
five-segment reusable solid rocket motors (RSRMV), as boosters. The results of RSRMV 
static test firings to date suggest an interaction between the booster and the thrust stand 
dynamics is likely compromising test data, especially the critical thrust rise rate, during 
motor ignition. Also there is a need accurately characterize the RSRMV ignition transient to 
provide high-fidelity inputs for SLS launch-induced environments simulations. Multiple 
time accurate, 3D computational fluid dynamics (CFD) simulations have been performed 
using the Loci-Chem code on NASA’s supercomputing resources to model the first half- 
second of the RSRMV ignition transient to address both of these issues. 

Project Details 

Time-accurate, fully 3-D CFD simulations of the RSRMV ignition transient were performed 
using the Loci-Chem code. A transient, pressure vs. time boundary condition was applied to 
the motor igniter. The simulations incorporated an updated Loci-Chem ignition module 
that solves the 1-D heat equation into the propellant grain to determine when each 
individual propellant surface cell should ignite. When ignited, the propellant "burns” using 
an empirical boundary condition based on propellant formulation and local pressure. The 
combustion products were modeled as a homogenous, single species gas. 



Validation efforts have included an input sensitivity study in which different boundary 
conditions were applied based on parametric values for inputs such as burn-rate and gas 
properties. Grid and time step parametrics have been done for solution verification. The 
grid sensitivity study used meshes ranging from 90-250 million total cells. The time-step 
study used time-steps ranging from 1-5 microseconds. 

Results and Impact 

CFD head end pressure results and thrust calculations match very well with static test data. 
Animations were created to facilitate identification of the fluid mechanisms that affect the 
motor pressure rise rate. Since the CFD simulations do not factor in thrust stand dynamics, 
the simulations are being used to identify thrust stand dynamics contributions to the 
measured thrust. 

Over 100 numerical probes were placed in a plane located at the entrance of the nozzle to 
capture the fluid properties of the gas at regular time-intervals. These were used to create a 
3-D fluid dynamics profile for an input boundary condition for additional CFD simulations 
for SLS launch-induced environments. 

Role of High-End Computing Resources and Services 

Utilizing NASA’s Pleaides supercomputer cluster, has allowed execution of multiple cases 
simultaneously. The performance of Loci-Chem scales very efficiently as the number of 
processors is increased. Using 2,400 processors per simulations allows completion in 2 
weeks. This turnaround time enables support of RSRMV and SLS during the design and 
development phases. NASA’s LOU archival system has also been necessary to store the 
several terabytes of data produced by each simulation. 

Philip Davis, NASA Marshall Space Flight Center 
philip.a.davis@nasa.gov 


Demo 10 

Solid Rocket Booster Ignition Overpressure Simulations for the Space Launch System 

Overview 

During launch, the Space Launch System (SLS) vehicle now in development will experience 
a series of large pressure waves emanating from the flame trench below which are caused 
by the fast ignition transient of the solid rocket boosters (SRBs), and have the ability to 
cause damage to the vehicle. While this ignition overpressure (IOP) phenomenon was 
observed during the Space Shuttle program, the existing engineering-level IOP analysis 
tools could not account for the different configuration of the SLS vehicle and launch pad. 

To study the effects of these differences, we performed time-accurate computational fluid 
dynamics (CFD) simulations of the SLS IOP event using NASA’s supercomputing resources. 

Project Details 

The goal of this work was to resolve multiple uncertainties which existed in the IOP 
analysis, including: the interaction of the SRB IOP with the core stage liquid engines at full 



power, effects of the new SLS mobile launcher design, and possible design changes to the 
flame deflectors below the vehicle. 

Several simulations were performed with varying levels of fidelity in order to specifically 
address each of these uncertainties and include updated vehicle and mobile launcher 
designs. Between 50 million and 150 million computational cells were used to model the 
detailed structures of the SRB and core stage plumes within the complex launch pad 
geometry. The simulations were realized with a CFD code named Loci/CHEM, using 
thousands of processors in parallel. 

Results and Impact 

Of the various uncertainties addressed in this work, the analysis of our simulations showed 
that the interaction of the SRB and core stage plumes had the most significant impact on 
the IOP environment. Because the core stage engines are at full power at the time of SRB 
ignition, the IOP wave is partially entrained or dissipated by the core stage plume flow field 
which creates smaller, less coherent waves with lower amplitude. In turn, the IOP wave 
interaction with the vehicle was more localized and peak impingement pressures were 
reduced. 

These and other findings were used to improve the IOP analysis which is provided to the 
SLS vehicle designers by removing unnecessary levels of conservatism. In addition, the 
higher fidelity simulations were used to modify parameters in the IOP analysis tools such 
that the predicted IOP waveform was reflective of the SLS vehicle and mobile launcher 
configuration, rather than being anchored to early Space Shuttle data. 

Role of High-End Computing Resources and Services 

Due to the time-sensitive nature of producing the IOP analysis for the SLS vehicle design 
cycles, the use of NASA’s supercomputing resources was vital to completing this work. 

Each simulation was run with thousands of processors on the Pleaides cluster, required 7 
to 15 days to complete, and produced over 2 terabytes of data. 

Researcher and Presenter atSC12: 

Brandon Williams, Marshall Space Flight Center 
brandon.williams@nasa.gov 


Demo 11 



Presenter at SC12: Christoph Brehm, NASA Ames 
Research Center, christoph.brehm@nasa.gov 

Demo 12 

Modeling and Simulation Support for NASA's 
Next-Generation Space Launch System 

Overview 

Computational Fluid Dynamics (CFD) simulations 
are being performed to support the design of NASA’s 
next generation launch vehicle, the Space Launch 
System (SLS). Modeling and simulation support 
includes characterizing the aerodynamic 
performance of the vehicle for a suitable ascent 
trajectory, determining the distributed aerodynamic 
line loads along the vehicle for structural analysis, 
and providing surface pressure signatures to assist 
in venting design for components of the vehicle. 

Project Details 

During the early design of the SLS, focus is placed on 
the stability and control of the vehicle during ascent 
and maintaining structural integrity throughout the 
mission. Characterization of the aerodynamic 
environments plays a key role in providing the 
necessary inputs for the Guidance Navigation and 
Control and the Structural Loads groups. In order to 
determine these environments before the vehicle is 
build, CFD simulations of SLS are performed at 
select points over the ascent trajectory and multiple 
angles of attack. The integrated and distributed 
aerodynamic loads on the vehicle are then extracted 
from the CFD solution and provided as input data 
for trajectory and structural analysis which are used 
to improve the design of the vehicle. 

In addition to providing information to modify the 
exterior shape and the interior structural 
components of the SLS, surface pressures over 
specific components of the vehicle are extracted and 
provided to the Venting group. This data is then 
used to determine were proper venting should 
occur along the vehicle. 

Results and Impact 

CFD simulations have been used to aid in critical 
design decisions for SLS early in the design cycle 
when wind tunnel data was not yet available. The 


CFD Simulations to Support the Next Gener 
Pads 

Overview 

With the retirement of the Space Shuttle, NASi 
new heavy-lift capability, the Space Launch Sy 
addition, commercial vehicles such as SpaceX’ 
Liberty and ULA’s Delta IV Heavy and Atlas V I 
being considered for transporting crew/cargo 
International Space Station (ISS). To accommc 
these vehicles, Kennedy Space Center (KSC) is 
their launch site capabilities. Components of p 
are the Flame Trench and the Main Flame Defl 
whose main purpose is to safely deflect the ex 
away from the vehicle as it lifts off. 

Project Details 

Re-design of the Main Flame Deflector (MFD) 
assessment of its performance regarding cont; 
plume in the trench, pressure and thermal loa 
and the surrounding environment for the laun 
commercial vehicles. A Computational Fluid D 
approach is well suited to provide the necessa 
capabilities during a rapid design cycle. Throu 
examination of the concern areas, incremental 
studies and comparative studies of various MI 
be achieved in a time and cost efficient manne 
consists of a CFD effort undertaken by the App 
and Simulation branch at NASA Ames Researc 
support of the MFD re-design project. 

Results and Impact 

This project assesses candidate MFD designs \ 
vehicle configurations and layouts and providi 
during the design cycle. Specifically, analysis ii 
containment characteristics, time evolution of 
temperature and heat transfer rate distributio 
trench surfaces. 

Role of High-End Computing Resources and Se 
Each of these simulations require roughly a wi 
time using up to 960 Westmere cores on NAS’: 
supercomputer. Since the simulations are unsi 
terabytes of data is generated for each calcula 
analysis and visualization is conducted using 1 
performance systems, including in-house part 
and other software. 


Researcher: Michael Barad, NASA Ames Researt 



ongoing modeling and support effort is making a large impact on import design decisions 
including: 

• Down selecting from several vehicle shapes 

• Outer Mold Line (OML) of the vehicle 

• Structural analysis of the core-stage and protuberances 

• Core-stage main engine hinge moment analysis 

CFD support for SLS will continue to advance as the vehicle matures and will make major 
impacts on: 

• Booster separation analysis 

• Core-stage fairing and main engine layout 

• 

Role of High-End Computing Resources and Services 

NASA’s Advance Supercomputing facilities enable fast and efficient turnaround time for 
CFD simulation support of SLS. The Pleiades machine allows viscous databases including 
several hundreds of simulations to be completed in under a week using 300 cores and 40 
hours of wallclock time per simulation. 

Researchers: Jeffrey Housman and Cetin Kiris, NASA Ames Research Center 
jeffrey.a.housman@nasa.gov and cetin.c.kiris@nasa.gov 
Presented at SC12: Cetin Kiris, cetin.c.kiris@nasa.gov 
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Demo 13 

Simulating Planetary Entry Environments for Space Exploration Vehicles 

Overview 

One of the challenges in designing a spacecraft for planetary entry is to predict the heating 
on the surface of a vehicle. It is important that designers have an accurate estimate of the 
aerothermal environment so they can select appropriate Thermal Protection System (TPS) 
materials and determine the TPS thickness required to protect a spacecraft from its high- 
speed entry into a planet’s atmosphere. Since it is impossible to reproduce the exact flight 
conditions with ground-based experiments, we must rely on computer simulations to 
predict worst-case thermal environments and to extrapolate test data to flight-like 
conditions. 

Project Details 

For the Mars Science Laboratory (MSL) and the Orion Multi-Purpose Crew Vehicle (MPCV) 
projects, Computational Fluid Dynamics codes were used to estimate the aerothermal 
environments at flight and at ground-based test conditions. These simulations, along with 
experimental data, were used to verify the accuracy of our computer models, to establish 
uncertainties of the heating estimates, and to study differences of various turbulence 
models. 


Results and Impact 


Our research revealed that even on a smooth body, it is difficult to accurately predict the 
heating on the separated backshell; therefore, large safety margins are needed when 
designing the backshell TPS. Simulations using a more complex two-equation Shear Stress 
Transport turbulence model did not perform better than the algebraic Baldwin-Lomax or 
the one-equation Cebeci-Smith turbulence model. 

Role of High-End Computing Resources and Services 

Since a typical simulation required more than 70 hours using 1,000 Pleiades cores, it is 
unfeasible to run hundreds of these calculations without the high-end computing resources 
at NAS. NASA’s Pleiades supercomputer played a critical role in the generation of 
aerothermal databases for the MSL and Orion MPCV programs. 

Researchers: Chun Tang, David Saunders, and Steven Yoon, NASA Ames Research Center 
Presen ter at SCI 2: 

Steven Yoon, NASA Ames Research Center 
s.yoon@nasa.gov 

Demo 14 

NASA Center for Climate Simulation Highlights 

Overview 

The NASA Center for Climate Simulation (NCCS) offers integrated supercomputing, 
visualization, and data interaction technologies to enhance NASA's weather and climate 
prediction capabilities. It serves hundreds of users at NASA Goddard Space Flight Center as 
well as other NASA centers, laboratories, and universities across the U.S. 

Project Details 

Over the past year, NCCS has continued expanding its data-centric computing environment 
to meet the increasingly data-intensive challenges of climate science. We doubled our 
Discover supercomputer’s peak performance to more than 800 teraflops by adding 7,680 
Intel Xeon Sandy Bridge processor-cores and, most recently, 240 Intel Xeon Phi Many 
Integrated Core (MIC) co-processors. 

A supercomputing-class analysis system named Dali gives users rapid access to their data 
on Discover and to high-performance software, including the Ultrascale Visualization 
Climate Data Analysis Tools (UV-CDAT). Analysis and visualization interfaces range from 
user desktops to a 17- by 6-foot hyperwall. NCCS is also exploring highly efficient climate 
data services and management with a new Hadoop Map/Reduce cluster while augmenting 
its data distribution to the science community. 

Results and Impact 

Using NCCS resources, NASA completed its modeling contributions to the 
Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report this summer 
as part of the ongoing Coupled Model Intercomparison Project Phase 5 (CMIP5). Ensembles 
of simulations run on Discover reached back to the year 1000 to test model accuracy and 
projected climate change through the year 2300 based on four different scenarios of 
greenhouse gases, aerosols, and land use. 



The data resulting from several thousand IPCC/CMIP5 simulations — as well as a variety of 
other simulation, reanalysis, and observation datasets — are available to scientists and 
decision makers through an enhanced NCCS Earth System Grid Federation Gateway. 
Worldwide downloads have totaled over 110 terabytes of data. 

Additional science highlights include a 10-kilometer-resolution simulation of global 
aerosols, modeling support for flight-based field campaigns by NASA and collaborating 
agencies, and study of the factors leading to spring 2011 extreme weather events including 
floods and tornadoes. 

Role of High-End Computing Resources and Services 

For their IPCC simulations, NASA’s Goddard Institute for Space Studies and Global Modeling 
and Assimilation Office used up to 160 concurrent jobs and 18,000 cores on Discover to 
simulate the breadth of CMIP5 scenarios. In aggregate, they completed tens of thousands of 
simulation-years and consumed more than 300 million Intel Xeon core-hours. 

Researcher and Presenter atSC12: 

Phil Webster, NASA Goddard Space Flight Center 
phil.wehster@nasa.gov 

Demo 15 

Ultrascale Climate Data Visualization and Analysis 

Overview 

Earth system scientists are being inundated by an explosion of data generated by ever- 
increasing resolution in both global models and remote sensors. Advanced tools for 
accessing, analyzing, and visualizing ultra-scale climate data are required to maintain rapid 
progress in Earth system research. To meet this need, the NASA Center for Climate 
Simulation (NCCS) at Goddard Space Flight Center is developing an advanced 
computational infrastructure that can provide high-performance analysis and visualization 
capabilities to the desktops of climate scientists. 

Project Details 

In collaboration with the Ultra-scale Visualization Climate Data Analysis Tools (UV-CDAT) 
development consortium, NCCS is developing climate data analysis and visualization tools 
for UV-CDAT, which provides data analysis capabilities for the Earth System Grid (ESG). 
These tools feature workflow interfaces, interactive 3D data exploration, hyperwall and 
stereo visualization, automated provenance generation, and parallel data processing. 
NASA’s DV3D is a UV-CDAT package that enables exploratory analysis of diverse and rich 
data sets stored in the Earth System Grid Federation (ESGF). DV3D provides user-friendly 
workflow interfaces for advanced visualization and analysis of climate data at a level 
appropriate for scientists. 

Results and Impact 

These computational services allow unprecedented access to high-performance data 
processing applications in support of Earth system science, facilitating climate data 



analysis operations that would otherwise be very difficult or impossible. This effort directly 
supports NASA’s mission to use remote sensing data and global models to better 
understand the dynamics of the Earth system. Analysis of observational and model data is 
essential in understanding how changes to Earth system processes, such as the climate, will 
impact humankind. 

Role of High-End Computing Resources and Services 

DV3D’s seamless integration with CDAT’s climate data management system (CDMS) and 
other climate data analysis tools provides a wide range of climate data analysis operations, 
e.g. simple arithmetic operations, regridding, conditioned comparisons, weighted averages, 
various statistical operations, etc. Several teams are developing parallel versions of these 
tools that will enable users to analyze and display large data sets that cannot currently be 
processed with existing desktop tools. This enables scientists to run analyses that were 
previously intractable due to the large size of the datasets and, using DV3D, seamlessly 
couple these analyses with advanced visualization methods. 

Researcher and Presenter atSC12: 

Thomas Maxwell, NASA NCCS 
thorn as. m axwell@ n asa.gov 


Demo 16 

NASA Climate Simulations and Observations for the IPCC and Beyond 

Overview 

Exploiting more than 30 years of global satellite observations, NASA scientists are creating 
observation-driven climate simulations that improve our insights into both climate model 
behavior and the complex physical processes and feedbacks that influence the Earth’s 
climate. 

NASA’s near- and long-term climate simulations and observations are contributing to 
international efforts such as the Coupled Model Intercomparison Project Phase 5 (CMIP5) 
and the Fifth Assessment Report of the International Panel on Climate Change (IPCC AR5), 
in addition to the US’s monthly National Multi-Model Ensemble (NMME) and the next 
quadrennial National Climate Assessment (NCA) in 2013. 

Project Details 

NASA’s global climate simulations and satellite observations allow us to explore evolution 
and variability of the Earth system from a variety of perspectives and time scales. 

• Researchers at the Goddard Institute for Space Studies (GISS) focus on long-term 
climate simulations from decades through centuries using their ModelE General 
Circulation Model (GCM) coupled with the Russell or HYCOM ocean model. 

• For nearer-term seasonal to decadal climate time scales, the Global Modeling and 
Assimilation Office (GMAO) applies its GEOS-5 Atmosphere-Ocean GCM. 

• To advance use of climate simulation results and related satellite observations and 
analyses, NASA serves the data via Earth System Grid Federation (ESGF) nodes on 
http:/ / esgf.nccs.nasa.gov/ and http:/ / esg-datanode.jpl.nasa.gov/. 



GISS’s Panoply tool (http://www.giss.nasa.gov/tools/panoply/) allows researchers to 
quickly visualize NASA’s climate simulation data, while NASA’s collaborations with the UV- 
CDAT project (http://uvcdat.llnl.gov/) support analysis capabilities such as interactive 3-D 
data exploration. 

Results and Impact 

NASA’s historical climate hind-casts and observation-driven near- and long-term 
simulations contribute to scientists’ understanding of the complicated interactions of 
climate’s physical processes. NASA’s collections of satellite observations since 1979 
provide scientists with essential data for analysis and comparison with model results. 
Taken together, the analysis of these near- and long-term climate observations and 
simulations significantly improves our understanding of Earth’s climate and climate 
models, and the resulting range of future climate change estimates are of great value for 
IPCC AR5 investigators, other scientists, and policymakers considering possible 
consequences. 

Role of High-End Computing Resources and Services 

GISS and GMAO climate simulations run on the Discover supercomputer at the NASA 
Center for Climate Simulation. From June 2010 through July 2012, climate processing has 
consumed more than 300 million core-hours and produced more than 60,000 simulation- 
years. 

Researcher and Presenter atSC12: 

Ellen Salmon, NASA Goddard Space Flight Center 
Ellen.Salmon@nasa.gov 

Demo 17 

Next-Generation Climate Data Services: MERRA Analytics 

Overview 

Scientific data services are critical to the mission of the NASA Center for Climate Simulation 
(NCCS). Modern Era Retrospective-Analysis for Research and Applications/Analytic 
Services (MERRA/AS) is a cyber infrastructure resource for developing and evaluating the 
next generation of climate data analysis capabilities. MERRA/AS supports 0bs4MIPs 
(observations for model inter-comparison projects) activities by reducing the time spent in 
the preparation of MERRA data used in data-model inter-comparison. It also provides a 
testbed for experimental development of high-performance analytics. 

Project Details 

MERRA/AS is a cloud-based service built around the Virtual Climate Data Server (vCDS) 
technology that NCCS currently uses to deliver Intergovernmental Panel on Climate Change 
(IPCC) data to the Earth System Grid Federation (ESGF). Crucial to their effectiveness, 
MERRA/AS’s servers use a workflow-generated realizable object capability to perform 
analyses over the MERRA data using the Hadoop/Map Reduce approach to parallel storage- 
based computation. 


Results and Impact 



The results produced by these operations will be stored by the vCDS, which will also be 
able to host code sets for those who wish to explore the use of Hadoop/Map Reduce for 
more advanced analytics. While the work described here will focus on the MERRA 
collection, these technologies can be used to publish other reanalysis, observational, and 
ancillary 0bs4MIPs data to ESGF and, importantly, offer an architectural approach to 
climate data services that can be generalized to applications and customers beyond the 
traditional climate research community. 

Role of High-End Computing Resources 

Powerful computing resources are necessary for on-demand analytic processing across 
30+ years of MERRA reanalysis data. The MapReduce operations leverage a cluster 
consisting of 36 Dell R710 servers each populated with twelve (12) 3-terabyte hard drives 
in addition to their operating system disks (CentOS 6.3). For interconnectivity, there is a 
36-port Fourteen Data Rate (FDR) InfiniBand (IB) switch plus a 48-port Gigabit Ethernet 
switch. Initial cluster metrics show one-way IB performance rates topping 6,000 
megabytes per second, peaks of 314 gigaflops per server, and an overall capability of 
approximately 11 teraflops. 

Researchers: 

Glenn Tamkin, NASA Goddard Space Flight Center, glenn.s.tamkin@nasa.gov 
John Schnase, NASA Goddard Space Flight Center, john.l.schnase@nasa.gov 
Dan DufJy, NASA Goddard Space Flight Center, daniel.q.duffy@nasa.gov 
Presen ter at SCI 2: 

Glenn Tamkin, NASA Goddard Space Flight Center, glenn.s.tamkin@nasa.gov 

Demo 18 

Recent Advances in High-Resolution Global Atmospheric Modeling 

Overview 

High-resolution global atmospheric modeling provides a unique tool to study the role of 
weather within the global climate system. The Goddard Earth Observing System Model, 
Version 5 (GEOS-5) is capable of simulating global weather at resolutions of 10 to 3.5 
kilometers (km) globally. These Global Earth-system Mesoscale Modeling (GEMM) 
simulations with GEOS-5 represent multiple scales of atmospheric phenomena — from 
convective clusters of deep convection, to hurricanes, to large mid-latitude storm 
systems — within a unified simulation of the global circulation. 

Project Details 

GEMMs with GEOS-5 produce a realistic representation of our atmosphere at resolutions 
comparable to many satellite observations. This provides an opportunity to perform 
comprehensive process studies on the formation of regional weather events and their 
impact on seasonal climate, improve the integration of existing observations into our data 
assimilation systems, and explore the potential of new observations through observing 
system simulation experiments. 

GEOS-5 uses the Goddard Chemistry Aerosol Radiation and Transport (GOCART) model 
within these GEMM simulations to include the global transport and direct radiative impacts 
of aerosols. These aerosols include dust particles lifted from the Earth’s surface by strong 



winds and deep convection, sea-salt blown from the ocean surface, plumes of sulfates from 
volcanic eruptions and fossil fuel emissions, and organic and black carbon within smoke 
from wildfires and human-initiated burning. These tiny particles can be transported large 
distances from their sources as they get caught up in the strong winds of the atmospheric 
circulation and have a significant impact on air quality, visibility, and human health. 

Results and Impact 

A 2-year GEMM simulation at 10-km resolution with GEOS-5 provides our first look at the 
potential of high-resolution global simulations integrating global weather phenomena 
within the variability of seasonal climate change. The 10-km GEMM simulation captured 
the dynamic variability of North Atlantic tropical cyclone activity for the very active 2005 
season and the more typical 2006 season. While resolution does improve the simulated 
variability of tropical cyclone formation, it is also a key element in improving the intensity 
of tropical cyclones that form within the model. At 10-km resolution the GEOS-5 GEMM 
simulations begin to represent the strongest hurricanes with near surface winds in excess 
of 160 miles per hour. 

Role of High-End Computing Resources and Services 

The 2-year, 10-km GEOS-5 GEMM simulation ran on 3,750 processors of the Discover 
supercomputer at the NASA Center for Climate Simulation for several weeks and produced 
over 400 terabytes of data. Experimental versions of GEOS-5 at 3.5-km global resolution 
represent the future of global atmospheric modeling including non-hydrostatic dynamics, 
cloud microphysics, and interactive aerosols (known within the community as global 
cloud-resolving models). This class of global models requires more than 20 times the 
computational capability of the Discover supercomputer to complete a 5-day forecast 
within the operational requirements of our data assimilation system. 

Researcher and Presenter atSC12: 

William Putman, NASA Goddard Space Flight Center 
William.M.Putman@nasa.gov 



Demo 19 

Causes and Consequences of Turbulence in the Earth's Protective Shield 

Earth and other planets are engulfed in the Sun’s atmosphere, which extends to about 10 
billion miles! The Earth’s dipole magnetic field shields us from most of the Sun’s effects and 
its frequent storms. However, the shielding is not perfect, and the solar wind is able to 
penetrate ("crack”) the magnetosphere. 

Our petascale kinetic simulations are critical to the development of a more complete 
understanding of the cause and conditions for the development of this crack in the 
magnetosphere, which can affect the Earth and its technological systems, and has caused 
over $4 billion in satellite losses alone. In this project we have: 

Used simulations with over 3 trillion particles to study the causes and consequences of 
turbulence in the magnetosphere 

Discovered mechanisms for the generation of the turbulence. 

Used 3D global simulations to examine the global consequences of the turbulence. 

Achieved closure on a number of longstanding issues on this topic, such as the role of 
certain type of waves in reconnection 

Confirmed several of our predictions through comparison with spacecraft observations 

Petascale simulations on massively parallel systems such as NASA’s Pleiades 
supercomputer have been crucial to our breakthrough studies and are enabling closure on 
critical issues in magnetospheric physics. 

Researchers: 

Homa Karimabadi, Tamara Sipes, University of California, San Diego/SciberQuest, Inc. 

Presen ter at SCI 2: 

Ari Le; University of California, San Diego/SciberQuest, Inc. 

Demo 20 

NASA Earth Exchange (NEX): A Collaborative Supercomputing Platform 

Overview 

The NASA Earth Exchange (NEX) represents a new collaboration platform for the Earth 
science community that provides a mechanism for scientific collaboration and knowledge 
sharing. NEX combines state-of-the-art supercomputing, Earth system modeling, workflow 
management, and NASA remote sensing data feeds to deliver a complete work environment 
in which users can explore and analyze large datasets, run modeling codes, collaborate on 
new or existing projects, and quickly share results among the Earth science communities. 

Project Details 

The main goal of NEX is to enable enhanced and more efficient use of Earth observations 
for NASA Earth science technology, research and applications programs. 



NEX utilizes the Pleiades supercomputing platform together with over 1PB of satellite, 
climate and model datasets in order to accelerate research in the Earth sciences. The 
results and project details can then be shared with the community through the NEX portal 
- a web-based knowledge network. Finally, NEX capabilities include a state-of-the-art 
collaboration facility that enables distributed teams to work together and share with the 
rest of the community through seminars and other outreach activities. 

Results and Impact 

As the development of NEX continues, it strives to lower the barrier of entry to data- and 
computer-intensive science. NEX will provide a mechanism for continuous engagement 
among members of the global science communities to work together to address grand 
challenge problems in Earth sciences. 

Role of High-End Computing Resources and Services 

The Pleiades supercomputing architecture combined with the massive data store and high- 
speed network enables NEX to engage large scientific communities and provide them with 
capabilities to execute modeling and data analysis on a grand scale, which was not 
previously achievable by most scientists. In one of the recent application of NEX, a team of 
researchers from NASA and South Dakota State University used the system to derive a new 
global vegetation product from over 80,000 Landsat scenes in order to provide a dynamic 
view of changing vegetation over the course of the year at 30-meter resolution. The 
processing of nearly 5 trillion pixels took only few hours on NEX, providing an 
unprecedented monitoring capability at a global scale. 

Researcher: Petr Votava, NASA Ames Research Center/CSUMB 

petr.votava@nasa.gov 

Researcher and Presenter atSC12: 

Dr. Ramakrishna Nemani, NASA Ames Research Center 
ramakrishna.nemani@nasa.gov 

Demo 21 

Powering Deep Space Missions: Thermoelectric Properties of Complex Materials 

Overview 

Thermoelectric materials are used to generate electric power from the decay of 
radioisotopes in deep space mission spacecrafts. Over the past decades only very simple 
materials (such as silicon and bismuth telluride) have been used for space application but 
experimentalists at JPL are studying more complex materials (such as La3Te4, zintl, and 
skuterrudites) with better efficiency. The objective of this project is to use state-of-the-art 
electronic structure calculations to compute thermoelectric properties of complex 
materials and guide the synthesis in the laboratory of materials with improved 
thermoelectric properties. 

Project Details 

The figure of merit for thermoelectric materials is proportional to the square of the 
Seebeck coefficient, which measures the electric field that arises in a material in response 



to an applied temperature gradient. This work focuses on computing the Seebeck 
coefficient for complex materials using first principles electronic structure calculations. 

The first principles electronic structure is obtained from the density functional theory, 
where the many body interaction between the electrons is approximated by a one-electron 
potential. The electronic structure is used to compute the Seebeck coefficient from a 
solution of the Boltzmann transport equation. 

We use the community software package Quantum Espresso (www.quantum-espresso.org) 
to compute the electronic structure. The computationally most costly numerical algorithms 
in Quantum Espresso are FFTs and orthogonalizations of large vector sets. 

Results and Impact 

We calculated the band structure, electronic density of states and Seebeck coefficient for 
La3Te4 and several zintl materials. We validated our modeling approach by comparing our 
results with published calculations and with measured data when available. We found 
excellent agreement. 

We found that the substitutions of appropriately chosen elements at the atomic level can 
alter the Seebeck coefficient. These exploratory studies have prompted new experiments 
that are expected to lead to the discovery of materials with improved figure of merit, as 
predicted by our calculations. 

Role of High-End Computing Resources and Services 

A typical first principles electronic structure calculation for complex materials takes up to 6 
weeks of computing time up to 200 processors. The memory requirements are about 439 
MB per process and 589 GB of disk space. These calculations cannot be performed on 
desktop computers, and supercomputing resources are essential. 

Researchers: Trinh Vo and Paul von Allmen, Jet Propulsion Laboratory, pva@jpl.nasa.gov 
Presenter at SC12: Heidi Lorenz-Wirzba, Jet Propulsion Laboratory, 
heidi.lorenz-wirzba@jpl.nasa.gov 

Demo 22 

Meeting NASA's High-End Computing Goals Through Innovation 

Overview 

Located at the NASA Advanced Supercomputing facility between San Francisco and San 
Jose, California, the High End Computing Capability (HECC) Project provides NASA 
scientists and engineers with a world-class computing environment to facilitate advances 
in their areas across all NASA technical mission directorates. The HECC project’s computing 
environment and full range of supporting services is focused on enabling discovery across 
a wide array of disciplines. Over the past year, HECC expanded Pleiades to over 1.7 PF, 
tripled the archive storage capability to over 90 PB, expanded the available tool set for 
diagnosing application performance and improved the ability of the security software to 
detect and prevent hostile actions taken against the computer environment. 

Project Details 

Since it’s inception in 2004, HECC has maintained a computing environment balanced by a 
strong set of related services. This combination of hardware supported by the services 



enables exciting innovations in science and engineering. The HECC hardware expanded in 
2012 reaching a theoretical peak processing capability exceeding 1.7 PF on Pleiades and 
tripled the archive capability to over 90 petabytes. Maintaining the capability of large 
shared-memory application development with single-system images of up to 4 terabytes 
and providing an internationally acclaimed data analysis and visualization capability 
providing both concurrent and traditional visualization capabilities joins with the additions 
to provide a unique discovery environment. 

The services attached to the facility provide a robust discovery environment focused on the 
success of our user community. This past year has seen improvements across the board 
with a strong focus on more useful tools, a strong outreach through a series of technical 
webinars, and a website designed to address our users challenging needs. 

Results and Impact 

HECC has been successful in providing a cost-effective, productive compute environment. 
The facility has been able to support a wide variety of users with a broad range of job 
requirements. The successes have ranged from supporting current flying missions such as 
Kepler and UAVSAR to research into the formations of the universe. 

Many of the key successes enabled through the HECC program are being highlighted in 
other areas of our booth at SC12. The successes highlighted by this project are the 
installation and integration of the new Sandy Bridge nodes adding FDR into the InfiniBand 
fabric, increasing the hypercube to a twelfth dimension and the growth of the archive. 

Role of High-End Computing Resources and Services 

The HECC project’s role is to provide all of NASA with a cost-effective, reliable, state-of-the- 
art supercomputing capability. Through strong relationships with the vendor community 
and a strong technical team, HECC continued its achievement of this role in 2012. 

Researcher and Presenter atSC12: 

William Thigpen, NASA Ames Research Center 
William.w.thigpen@nasa.gov 

Demo 23 

Continuous Enhancements to the Pleiades Supercomputer for Maximum Uptime 

Overview 

When Pleiades was initially installed in 2008, the system was a homogeneous cluster with 
51 thousand cores and Double Data Rate InfiniBand. Since then, the system has been 
increased by nearly 2.5x to 126 thousand cores with 4 different generations of processors 
and 3 generations of (DDR, QDR, and FDR) InfiniBand in a single cluster. This has 
challenged the HECC systems team to develop tools and processes to maintain a productive 
operational supercomputer with minimal downtime. 

Project Details 

Operating an InfiniBand cluster of nearly 12,000 nodes requires the HECC systems team to 
be able ensure that the InfiniBand interconnect for the system is robust to handle fault 
conditions and to be able to recover gracefully. One of the enhancements developed was a 
modification to the InfiniBand Subnet Manager to enable the live integration or removal of 



nodes from the InfiniBand fabric. This ability allowed the system to continue to provide 
computing cycles at the same time that resources added to the system or removed for 
maintenance activities. 

Another enhancement to the system was feature of suspend and resume of the batch 
scheduler software. By stopping jobs on the system, this enabled some maintenance 
activities without needing to drain the system for maintenance time, which is an expensive 
process in terms of lost computing cycles. By suspending jobs on the system, the overhead 
maintenance time can be greatly reduced with minor impact on the users. 

A related improvement to the batch scheduler is the ability to have a malleable walltime for 
batch jobs. For times when suspending jobs is not viable and the draining of jobs on the 
system is required, the ability to run jobs as close as possible to the maintenance window is 
desirable. By specifying a minimum walltime and a maximum walltime, the batch 
scheduler will allow jobs to run if the minimum walltime can be started. This enables more 
efficient use of the system. 

Results and Impact 

By developing enhancements to the batch scheduler and InfiniBand software, the HECC 
systems team is able to maximizing the computing cycles available for NASA missions. 

Researcher and Presenter atSC12: 

Davin Chan, NASA Ames Research Center 
davin.chan@nasa.gov 


Demo 24 

Live Demonstrations of 100-Gbps File Transfers Across LANs and WANs 

Overview 

Overcoming throughput bottlenecks in large file transfers over wide area networks 
(WANs) is critical to sharing data from NASA's Earth Science programs. The High-End 
Computer Network (HECN) Team at NASA Goddard Space Flight Center is investigating 
new high-performance servers, disk configurations, and 40- and 100-gigabit per second 
(Gbps) network interfaces to resolve these bottlenecks. 

In cooperation with several partners, NASA will conduct live, real-time demonstrations of 
100-Gbps file transfer experiments between the NASA exhibit (831) at SC12 and NASA 
Goddard in Greenbelt, MD, across 100-Gbps local and wide area networks. 

Project Details 

New technologies featured in these demonstrations include: 

Intel Sandy Bridge EP processors and Supermicro X9DR3-F PCI Express Gen3-based 

motherboards with quad channels to memory 

OCZ Vertex3 solid-state disks and LSI MegaRaid 9271 RAIDs 

Hot Lava three-port 40-Gigabit Ethernet (GE) network interface cards 

Prototype 100GE network interface cards developed by Acadia Optronics through a 

Department of Energy grant 

lOOGE-capable switches and optical gear from Ciena, Brocade, 

Infinera, ADVA, Fujitsu, Alcatel-Lucent, Arista, and possibly others 



Results and Impact 

Science research has transitioned from relying on two classic building blocks — theory and 
experimentation — to also using modeling and simulation. Increasingly, these techniques 
require massive amounts of data, which must be gathered, analyzed, stored, and 
transported among multiple sites worldwide. 

The High-Performance Networking for Petascale Science Consortium was formed to 
investigate, evaluate, develop and demonstrate new techniques and methods to create 
services for transporting extremely large-scale data streams in support of petascale science 
at 100 Gbps. The consortium includes NASA Goddard, the International Center for 
Advanced Internet Research at Northwestern University, the Mid-Atlantic Crossroads, the 
Laboratory for Advanced Computing at the University of Chicago, Johns Hopkins University, 
the Naval Research Laboratory, the StarLight International/National Communications 
Exchange Facility, and the Metropolitan Research and Education Network. Consortium 
support comes from the ESnet Advanced Network Initiative, Ciena, and Sidera Networks. In 
addition to the above-mentioned collaborators, other key demonstration partners are 
SCinet, National LambdaRail, Internet2, Brocade, Fujitsu, cPacket, Arista, and ForcelO. 

Role of High-End Computing Resources and Services 

Using NASA Goddard's Science and Engineering Network (funded by the NASA Center for 
Climate Simulation) and several R&D WANs managed by our partners, the HECN Team 
achieved a throughput rate of approximately 60 Gbps when copying files over WANs at 
SC11. With new PCI Express Gen3 servers and processors, we expect to demonstrate nearly 
100 Gbps cross-country WAN file transfers at SC12. 

Researcher and Presenter atSC12: 

Bill Fink, NASA Goddard Space Flight Center 
hill.fink@nasa.gov 

Demo 25 

Untangling the Computing Landscape for Climate Simulations 

Overview 

The NASA Center for Climate Simulations (NCCS) at Goddard Space Flight Center (GSFC) 
supplies high performance computing and data services engineered to support climate 
simulations within the NASA Science Mission Directorate. Within the past few years, a 
number of technologies have emerged with the potential to change the landscape of high 
performance computing for climate simulations. The NCCS, along with its user community, 
has been exploring NVidia Graphical Processing Units (GPUs), Intel Many Integrated Core 
(MIC), Cloud Computing, Virtualized Infiniband, and have even begun to explore non-X86 
processors such as the ARM or Atom. Making sense of these technologies and how to 
deploy them efficiently and effectively is not an easy challenge. This presentation will 
discuss the results from our exploration of a number of these advanced computing 
technologies and recommendations for how these can be deployed for high-end climate 
simulations. 


Project Details 



The NCCS high-end computing cluster, Discover, was augmented in October 2012 with Intel 
MIC processors. The upgrade consisted of 480 IBM iDataPlex nodes configured with Intel 
SandyBridge processors. Initially, the system has been configured with 240 Intel MIC 
processors with enough capacity to integrate more MIC processors in the future. 

The NCCS has been exploring the use of cloud technologies and Virtualized Infiniband to 
support climate simulations within the Discover cluster. A test system of eight nodes was 
created out of representative components from the Discover cluster and configured with 
both native operating systems and virtual hosts. Representative kernel and application 
level benchmarks were run on this test cluster. 

Also, the NCCS has begun exploring the use of non-X86 processors for potential use within 
climate simulations. This work is in the very beginning stages. 

Results and Impact 

The presentation will discuss a comparison of GPU to MIC processors and discuss how the 
NCCS is thinking about future deployments of these accelerator technologies. In addition, 
the presentation will show the kernel and benchmark results from the Virtualized IB test 
bed. Finally, the presentation will discuss future work on non-X86 processors and their 
potential to impact climate simulations. The overall impact of these technologies on high- 
end computing will be discussed. 

Role of High-End Computing Resources and Services 

The applications being run on Discover contribute to the overall scientific knowledge of 
climate change. Coupled with a robust and growing data analytic environment, the NCCS 
provides an environment in which large-scale climate models are executed, data is 
analyzed, and discoveries are made. 


Researcher and Presenter atSC12: 
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Demo 26 

Simulating Galaxies and the Universe 

Overview 

According to the now-standard Lambda Cold Dark Matter (ACDM) theory, most of the 
universe is made up of invisible dark matter and dark energy. Bolshoi and BigBolshoi are 
the most accurate high-resolution ACDM simulations of the large-scale structure of the 
universe. Galaxy formation within the ACDM framework is so complicated that observation, 
theory, and simulation must work together in order to achieve understanding. 

Project Details 

We are using our state-of-the-art hydrodynamics plus N-body code hydroART to simulate 
hundreds of cosmological regions, each containing a large galaxy and many of dwarf and 
satellite galaxies. 



The resulting large library of high-resolution cosmological simulations of galaxies forming 
in the standard ACDM cosmology, in different environments, and with different degrees of 
interaction provides an accurate sample of the observable universe. 

Our Monte Carlo radiative-transfer code, Sunrise, enables us to make realistic images of the 
simulated galaxies in many wavebands at many times during each simulation, including the 
important effects of stellar evolution and dust. Directly comparing a statistical sample of 
hundreds of such "simulated observations” with HST, Spitzer, Herschel and other 
observations will help to clarify how galaxies form in different environments. 

Since the resulting hundreds of thousands of images produced are far too many to be 
analyzed entirely by human eyes, we are currently developing techniques for human- 
computer interactions in order to classify these images using computer vision methods 
previously developed for other applications. Also 300,000 participants in the citizen 
science project GalaxyZoo are classifying real and simulated galaxies. 

Results and Impact 

Our simulations of galaxies forming in the standard ACDM cosmology are providing 
theoretical support for the interpretation of data gathered from NASA’s Hubble Space 
Telescope (HST) and other observatories, especially in connection with the largest HST 
program: Cosmic Assembly Near-IR Deep Extragalactic Legacy Survey (CANDELS). 

Role of High-End Computing Resources and Services 

The Columbia and Pleiades supercomputers at the NASA Advanced Supercomputing (NAS) 
Facility have been extremely helpful in running dissipationless simulations of large-scale 
structure evolution and the highest resolution (10-30 parsecs) hydrodynamic galaxy 
simulations. In addition, running our Sunrise "observations" of galaxy simulations on 
NASA’s new 64 Pleiades graphic processor unit (GPU) nodes is allowing a factor of ten 
speedup. Lastly, collaboration with NAS visualization experts has been crucial to visualizing 
and interpreting the results of these simulations, and in generating materials for the 
purposes of public outreach. 

Researcher: Joel Primack, University of California Santa Cruz 

joel@ucsc.edu 

http://hipacc.ucsc.edu 

Presenter at SC12: Nina McCurdy, University of California Santa Cruz 
nina.mccurdy@gmail.com 

Demo 27 

The Mysterious Origin of Stellar Masses 

Overview 

Most stars have a mass similar to that of our Sun or a few times smaller. Extreme cases, 100 
times less or more massive than the Sun, are much more rare. The classical theory of 
gravitational instability cannot explain this characteristic stellar mass, nor the wide spread 
of values between the extreme cases. 

The key to this problem is the turbulent nature of stellar nurseries, cold clouds formed 
primarily of molecular hydrogen, characterized by violent internal motions many times the 



speed of sound. Through the effect of strong shocks, this supersonic turbulence results in a 
complex network of dense filaments, where the observed stellar masses can be assembled. 
Using NASA’s fastest supercomputer, we carried out the largest simulations to date of star- 
forming clouds, capable of capturing the full range of stellar masses. 

Project Details 

We use a heavily modified version of the public adaptive-mesh-refinement code Ramses, 
allowing us to run effectively on a large fraction of the NASA/Ames Pleiades 
supercomputer. By focusing the computational resources onto the densest regions, with 
the adaptive-mesh-refinement method, we achieve a maximum resolution equivalent to 
that of a uniform grid with 131,0723 computational cells. 

Our simulations yield the full range of stellar masses, from a few percent of a solar mass, to 
almost one hundred solar masses. They also reproduce the observed probability 
distribution of stellar masses. 

Results and Impact 

Our simulations allow us to test theoretical models of star formation that are necessary to 
understand the formation of galaxies. They elucidate the process that assembles the mass 
of stars and shed new light onto the early phases of star and planet formation. 

Each simulation results in thousands of stars, providing very large statistical samples of 
initial conditions for protoplanetary discs around normal stars and gas accretion around 
massive stars. Such simulated initial conditions can be used to make predictions and 
interpretations for observations with space and ground based observatories. 

Role of High-End Computing Resources and Services 

Thanks to the NASA/Ames Pleiades supercomputer, we have achieved the largest start- 
formation simulation to date, using thousands of processors. NASA’s large shared-memory 
machines, such as Bridge3 and Bridge4, and huge online disk storage have provided us 
with a powerful environment for data analysis, crucial to extract scientific results from the 
simulations. 

Deeper insight on the structure and evolution of the modeled star-forming regions was also 
provided by graphic visualizations carried out by the visualization team at NASA Ames. 

Researcher and Presenter atSC12: 

Paolo Padoan, ICREA & ICC, University of Barcelona 
ppadoan@icc.ub.edu 

Demo 28 

Hot-Plasma Geysers on the Sun 

Overview 

The Sun is covered by geyser-like plasma eruptions ("spicules") which provide mass and 
energy to the million-degree corona and to the solar wind affecting the Earth's space 
environment. Due to the small-scale structure and fast dynamics these eruptions were not 
well resolved in observations, and their origin was not known. Realistic 3D radiative 
magneto-hydrodynamics (MHD) simulations allowed us to reproduce these eruptions, 



investigate their structure, and uncover the driving mechanism hidden in the turbulent 
convection zone below the visible surface of the Sun. 

Project Details 

The simulation domain includes the upper convective boundary layer, photosphere and 
chromosphere of the Sun. The mathematical model is based on first physical principles, and 
employs the Large-Eddy Simulation (LES) approach of MHD turbulence simulations and a 
multi-group approximation for radiation transfer. The simulations utilize a parallel 
"SolarBox" code developed at NASA/Ames. The code runs very efficiently on large number 
(1000-4000) of processors. 

Driven by the energy flux from the energy-generating core of the Sun the turbulent 
convection forms granulation cells with high-speed downflows in the intergranular lanes. 
The overturning and shearing flows below the solar surface form compact self-organized 
turbulent vortex tubes with high-speed swirling flow. The vortex tubes are concentrated in 
the intergranular lanes. They interact with surrounding magnetic field twisting magnetic 
field lines. Due to the pressure excess accumulated in the low atmospheric layers the 
magnetized vortex tubes erupt in a manner similar to geyser eruptions. 

Results and Impact 

The realistic simulation revealed that the ubiquitous flow eruptions on the Sun are 
produced by self-organized magnetized vortex tubes generated by turbulent convection 
below the visible surface of the Sun. These results allow us to better understand the mass 
and energy flowing from the Sun into the heliosphere and the Earth's space environment, 
and use this knowledge for interpretation of data from the Solar Dynamics Observatory 
(SDO), Hinode and IRIS missions, and for developing physics-based models of the solar 
dynamics and activity. 

Role of High-End Computing Resources and Services 

The realistic radiative 3D MHD numerical simulations require massive computational 
resources. These tasks were accomplished using the Pleiades supercomputer system at the 
NASA Ames Advanced Supercomputing (NAS) Division, which demonstrated a highly 
efficient performance for massive parallel computations. The 3D visualization (with the 
support of Tim Sandstrom and Chris Henze, NASA/NAS) greatly contributed to the success 
of this project. 

Researcher: Irina Kitiashvili, Stanford University 
irinasun@stanford.edu and Alan Wray, NASA Ames Research Center 
Alan. A. Wray@nasa.gov 
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Demo 29 

Turbulent Life of Kepler Stars 

Overview 

NASA's Kepler mission has discovered a whole new world of extra-terrestrial planets and 
variable stars. It was particularly surprising that the stars with convective envelopes much 



thinner than on the Sun show significant fluctuations in brightness, and also strong 
stochastic acoustic oscillations of the solar type. Because of the large distance the surface of 
these stars cannot be resolved in observations. Thus, numerical simulations is the only way 
to learn about the structure of their surface and dynamics. The simulations provided the 
first detailed view of the turbulent surface of the planet-hosting stars that are more 
massive than the Sun, and allowed us to discover the mechanisms of the stellar acoustic 
emission. 

Project Details 

The stellar turbulence simulations were performed using 3D radiative hydrodynamics 
code "StellarBox" developed at NASA/Ames by Alan Wray and Nagi Mansour. The code 
employs high-order Pade finite-difference scheme and LES (Large-Eddy Simulation) 
subgrid-scale models for accurate description of the stellar turbulence and acoustics. The 
simulations were performed for several Kepler-target stars in the mass range from 1.05 to 
1.7 solar masses, using the standard hydrostatic stellar evolutionary models for the initial 
conditions. The simulations were performed on the Pleiades system using 1,000-2,000 
processors. 

Results and Impact 

The simulations revealed that the surface convective layers on the stars are much more 
dynamic and turbulent than on the Sun. Because of their higher luminosity, the turbulent 
energy transport is substantially greater driving supersonic turbulence and strong acoustic 
emission. The stellar granulation cells are bigger than the solar granulation, and often 
penetrated by high-speed plumes rarely seen on the Sun. The most interesting 
phenomenon is formation of large-scale turbulent vortices on the star's surface, which 
generate acoustic waves. These waves form global resonant acoustic modes discovered for 
this type of stars by the Kepler mission. The strong turbulence and acoustic emission found 
in the simulations are likely to energize the stellar wind leading have profound effects on 
the extra-terrestrial planets. 

Role of High-End Computing Resources and Services 

The radiative hydrodynamics numerical simulations of the stellar turbulent dynamics were 
performed by Dr Irina Kitiashvili (Stanford) using the Pleiades supercomputer system at 
the NASA Ames Advanced Supercomputing (NAS) Division, which demonstrated a highly 
efficient performance for massive parallel computations. The 3D visualization was created 
by Tim Sandstrom and Chris Henze (NASA/NAS). 

Researchers: Alexander Kosovichev, Stanford University 
sasha@sun.stanford.edu and Nagi Mansour , NASA Ames Research Center 
Nagi.N.Mansour@nasa.gov 
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Demo 30 

Modeling Weather on the Sun 



Overview 

The Sun’s weather is modeled (in a similar fashion to Earth’s weather) by numerically 
solving the conservation equations for mass, momentum and energy plus the induction 
equation for the magnetic field. This allows us to study the magneto-dynamics of the solar 
surface, especially the emergence of magnetic fields in both the quiet Sun and active 
regions. It is magnetic fields emerging through the solar surface that control the heating of 
the solar chromosphere and corona and the eruption of flares and coronal mass ejections. 

Project Details 

We simulate magneto-convection near the solar surface using the 3D, compressible, 
magneto-hydrodynamic, finite difference STAGGER code in a Cartesian grid with the 
vertical coordinate stretched for finer resolution near the visible surface. A tabular real gas 
equation of state is used including ionization and excitation of hydrogen, helium and other 
abundant elements. Radiative heating and cooling is calculated by solving the non-grey 
radiation transfer equation with four opacity and emissivity groups using long 
characteristics. 

Uniform, untwisted, horizontal, one kilogauss magnetic field is advected into the domain 
through the bottom boundary at 20 Mm depth (10% of the geometric depth of the solar 
convection zone but 2/3 of its density scale heights) in order to represent the effects of 
dynamo action in the deeper layers. 

Results and Impact 

Magneto-convection, acting on large-scale dynamo produced fields from the lower 
convection zone produces a hierarchy of magnetic loops with smaller ones as serpentine 
features on larger loops. 

Large loop complexes emerge through the solar surface as active regions. First, mixed 
polarity fields emerge as small, granule sized loops over enlarged, elongated granules. 

Then opposite polarity fields collect into separate, unipolar concentrations driven by the 
underlying large scale field structure to form the active region. 

Validity of inversions of Stokes profiles to obtain 3D magnetic fields is being evaluated by 
comparing inversions of simulated Stokes spectra with actual magnetic fields in the 
calculations. Inversions of weak fields (< 100-200 G) are found to be unreliable due to the 
complex vertical structure of the magnetic and velocity fields and exacerbated by the 
limited resolution of the telescopes. 

These results enable us to understand and evaluate the results from NASA missions such as 
the Solar Dynamics Observatory and Hinode. 

Role of High-End Computing Resources and Services 

These simulations require several million processor hours on the NASA Pleiades 
supercomputer, which takes months of wall time even running on thousands of cores. 

They would be impossible without these resources. 

Researcher and Presenter atSC12: 

Robert Stein, Michigan State University 
stein@pa.msu.edu 



Demo 31 

Weather on Mars: The Meteorology of Gale Crater 

Overview 

On August 5th, 2012, the Mars Science Laboratory 'Curiosity' rover landed in Gale Crater 
near the martian equator. The chosen landing site was deemed both of high scientific value 
as well as safe for entry, descent and landing (EDL) and long-term surface operations. To 
demonstrate safe conditions in the martian atmosphere over Gale Crater, a long-term 
investigation into the meteorology around the site was undertaken. 

Project Details 

As a way of understanding the local meteorology at Gale Crater, we employed the Mars 
Weather Research and Forecasting (MarsWRF) general circulation model (GCM) to 
simulate atmospheric behavior at various times of year over Gale Crater. 

The flexible nature of MarsWRF allows us to use the same model architecture from the 
largest, global scales down to very fine resolution grids that capture the local behavior 
within the crater itself, including winds, temperature and pressure. Local topography and 
surface properties are obtained from orbital observations from previous spacecraft. 

These results can be compared to data obtained by the Curiosity rover during its mission to 
evaluate the efficacy of such GCMs to serve a predictive role in understanding local 
meteorology. Additional investigations, using a thicker, warmer atmosphere 
representative of early conditions on Mars can be run to explore the possibility of liquid 
water existing for long periods in this location much early in Mars’ history. 

Results and Impact 

Results with the MarsWRF GCM demonstrate a strong diurnal component to the wind field 
inside Gale Crater, with largely upslope winds (northerlies at the Curiosity landing site) 
during daytime and downslope winds (southerlies) at night. This is consistent with 
topographically driven circulation and is similar to behavior seen along the flanks of 
terrestrial topographic features. 

The results of this work, and, more specifically, identification of those areas in which the 
GCM does not match well to the observed meteorology by Curiosity will improve 
predictions for future landed missions on Mars and reduce overall system margins, which 
will have the effect of reducing mission cost, mass and complexity. 

Role of High-End Computing Resources and Services 

Use of the 'Cosmos’ supercomputer at JPL for the general circulation modeling, along with 
the availability of large, long-term storage of model output were invaluable resources for 
achieving these objectives. A typical 30-day simulation, with four embedded model nests 
(from global down to local scale) takes about three days on 16 CPUs. 

Researcher: Michael Mischna, Jet Propulsion Laboratory/California Institute of Technology, 
michael.a.mischna@jpl.nasa.gov 
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Demo 32 

Enhancing Performance of NASA's High-End Computing Applications 

Overview 

The Application Performance and Productivity (APP) group of the High-End Computing 
Capability (HECC) Project at the NASA Advanced Supercomputing facility has three key 
activities: enhancing performance of high-end computing applications, leveraging software 
technologies to improve user productivity, and characterizing performance of current and 
future architectures. 

Project Details 

Recent application optimization efforts boosted the performance of several user codes, 
including introducing OpenMP in two codes, Satellite Data Simulation Unit (SDSU) code and 
a solar acoustic computational code (A_SH_B_WAVES) to gain a performance boost of 8x 
and 2.7x respectively; and improving the 10 performance of the Goddard Cumulus 
Ensemble (GCE) code, allowing it to scale up to 4096 cores. 

Results and Impact 

The group also enhanced user productivity and system efficiency in several ways including 
developing a methodology to package hundreds to thousands of serial (UCLA- Aurora, JPL- 
L2_FP, GRC-SWBLI) or parallel (ARC -Aero CFD) simulations in a single batch job; creating a 
tool, lumber, to facilitate rapid analysis of individual job failures and system-wide issues by 
mining for error patterns in log messages; and developing a methodology, unit, to detect a 
large class of uninitialized floating-point variables, which are undetected by standard 
compiler options. 

Role of High-End Computing Resources and Services 

The APP group uses a suite of NASA-relevant codes regularly to characterize performance 
of new high-performance computing architectures, including new Intel Xeon-based 
systems, NVDIA GPUs and Cloud offerings. The group helped develop a knowledge base 
approach for user documentation and initiated user training through regular webinars. 

Researchers and Presenters at SC 12: 
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Demo 33 

Designing Curiosity's Perfect Landing on Mars 

Overview 

At 10:32 pm (PDT) on August 5th, 2012 millions of people from around the world watched 
as the Curiosity rover gently touched down onto the floor of Gale Crater next to a six- 
kilometer mountain. Five hundred-thousand lines of the most sophisticated interplanetary- 
autonomous-autopilot code brought the spacecraft down from a speed of 13,000 
kilometers per hour (km/h) at the top of the Martian atmosphere to 0 km/h on the surface 
of Mars in just seven minutes. NASA’s Jet Propulsion Laboratory (JPL) used its unique entry, 
descent and landing (EDL) knowledge combined with thousands of CPU hours from several 



Beowulf clusters to orchestrate the sequence of events that successful placed this 2,000- 
pound rover onto the surface of Mars. 

Project Details 

The Jet Propulsion Laboratory built the biggest robotic exploration rover yet, which is 
carrying 10 times more pay load in scientific instruments than the Mars Exploration Rovers. 
Curiosity is the most comprehensive Martian geologist to rove the red planet. It carries 17 
different cameras, 4 spectrometers, 2 radiation detectors, an environment sensor, and an 
atmospheric sensor. A radioisotope thermoelectric generator (RTG) that produces 110 
watts powers the rover. 

Engineers at JPL sent Curiosity on a nine-month journey across 350,000,000 miles for a 
precision landing on Mars. JPL designed the spacecraft to steer itself using a series of S- 
curves for a guided entry through Mars’ atmosphere. Three minutes from touchdown the 
spacecraft parachute-decelerated to about 200 mph in an atmosphere that is 100 times 
thinner than that of earth. Finally, the spacecraft used a series of retro rockets and a sky 
crane to complete its precision landing sequence touching down inside the smallest ever 
Martian landing ellipse. 

Results and Impact 

Curiosity’s overarching science objective is to evaluate whether Mars has ever had or still 
has the necessary environmental conditions for life. Scientists believe that the alluvial fan 
on the floor of Gale Crater may have been created by water-carried sediments and may 
hold the key to answering the question about life on Mars. Scientists also believe that 
Mount Sharp’s layered formation may contain minerals formed in water that are the 
building blocks of life. Curiosity has a long journey of discovery ahead of it, but it has 
already captivated the hearts of scientists, engineers, and many others through its 
revolutionary achievements in science and technology. 

Role of High-End Computing Resources and Services 

Sixteen months ago, JPL's Entry, Descent, and Landing team (EDL) began working with 
JPL’s Supercomputing team to create a fully customized supercomputing environment 
necessary to run the complex guided entry simulations. The Supercomputing team used 
version-control and automated build techniques to allow the team to recreate any version 
of the customized environment. 

The Monte Carlo simulation used for Curiosity ran a series of high-fidelity trajectory 
integrations that had a unique set of inputs for each embarrassingly parallel simulation. A 
precision landing with guided entry required each Monte Carlo simulation to have 8,001 
separate trajectory integrations, requiring 90 minutes of run-time for each simulation. 

JPL's Supercomputing capabilities allowed the EDL team to dramatically reduce the risk in 
landing at Gale Crater. "Having that resource available made it very nice for us”, stated Paul 
Burkhart from JPL EDL Guidance and Control Systems. 

Researcher: John Grotzinger, Jet Propulsion Laboratory/California Institute of Technology 
Presenter at SC12: Timofey Ovcharenko, Jet Propulsion Laboratory, CSC, tima(a)jpl. nasa.gov 



Demo 34 

The Search Continues: Kepler's Quest for Habitable Earth-Sized Planets 

Kepler is NASA’s first mission capable of finding Earth-size and smaller planets around 
other stars. Launched in March of 2009, the Kepler spacecraft has been continuously 
monitoring over 150,000 stars in the constellations of Cygnus and Lyra, watching for the 
periodic dimming of the stars that may indicate the presence of an exoplanet passing in 
front of the star from the perspective of our own solar system. 

Project Details 

Kepler's sole instrument is a specially designed photometer with a 0.95-meter aperture 
telescope. It has a very large field of view for an astronomical telescope — 115 square 
degrees, which is comparable to the area of your hand held at arm's length. It needs that 
large a field in order to observe the necessary large number of stars. 

Kepler's photometer contains the largest digital camera to ever fly in space, consisting of 42 
charge-coupled devices (CCDs) for a total of nearly 95 megapixels. Pixels of interest are 
downlinked once a month and transferred to the Kepler Science Operations Center (SOC) at 
NASA Ames Research Center in California. Here the pixels are calibrated, combined to form 
light curves, corrected for systematic errors introduced by the instrument, then searched 
for the signatures of transiting planets. 

Results and Impact 

As of February 2012, the Kepler team has found 2321 planetary candidates circling 1790 
host stars (with 77 confirmed to be planets by other means). Roughly a third or more are in 
multiple-planet systems. Some of these systems also involve planetary candidates orbiting 
binary star systems (circumbinary planets). Hundreds of candidates have now been found 
that are Earth-sized although not in the habitable zone of their respective stars. Several 
planetary candidates however do occupy the habitable zone, although they are not Earth- 
sized. As the mission continues, additional longer period candidates continue to be found 
and trends suggest that we should start filling in the most interesting portion of parameter 
space, namely, the habitable zone Earth-size planets. 

Role of High-End Computing Resources and Services 

We have successfully ported the most computationally expensive portions of the pipeline 
(Pre-Search Data Conditioning, Transiting Planet Search, and Data Validation modules) to 
the Pleiades supercomputer, and have demonstrated near perfect (linear) strong scaling up 
to 75,000 processors. We have also developed software that automatically copies data 
between the Kepler SOC and Pleiades so that we can run the less expensive parts of the 
pipeline in the SOC, while running the most expensive parts on Pleiades, in a manner that is 
transparent to the operators. The addition of features to other portions of the data 
processing pipeline has necessitated their migration to the supercomputer as well. That 
work is currently underway. Employing the supercomputer will allow us to realize the 
benefit of our code improvements on a much shorter time-scale than we currently operate 
under due to our locally limited computing resources. This will further benefit the general 
public as well since, beginning in December, both pipeline results as well as data will 
become available through NExScI as we continue on into Kepler’s Extended Mission. 
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